Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimization - deterministic deployments #131

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Optimization - deterministic deployments #131

wants to merge 3 commits into from

Conversation

alukach
Copy link
Collaborator

@alukach alukach commented Dec 24, 2024 •
edited
Loading

What I am changing

In this PR, I attempt to remove any changes in services that would trigger unnecessary updates to the deployed services (ie alterations to the service that do not affect the operation of the service).

How I did it

Remove timestamps from handler names and remove timestamps from zipped files.

Impact

Admittedly, the positive impact in regards to reducing deployment times was minimal. It did seem as though the amount of services updated did seem some reduction, however I was unable to get a strong read on how much time this may have saved in the face of total deployment time. I believe it's worth considering keeping this change all the same.

@github-actions GitHub Actions
Copy link

🍹 preview on cerulean-cloud-images/test

Pulumi report 
   Previewing update (test):
@ previewing update........

@ previewing update..............
   pulumi:pulumi:Stack cerulean-cloud-images-test running 
@ previewing update.....
   gcp:container:Registry cerulean-cloud-images-test-registry  
@ previewing update.....
   docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image  Building your image for linux/amd64 architecture.
~  docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image update [diff: ~build]; Building your image for linux/amd64 architecture.
   docker:index:Image cerulean-cloud-images-test-cr-offset-tile-image  Building your image for linux/amd64 architecture.
~  docker:index:Image cerulean-cloud-images-test-cr-offset-tile-image update [diff: ~build]; Building your image for linux/amd64 architecture.
   docker:index:Image cerulean-cloud-images-test-cr-tipg-image  Building your image for linux/amd64 architecture.
~  docker:index:Image cerulean-cloud-images-test-cr-tipg-image update [diff: ~build]; Building your image for linux/amd64 architecture.
   pulumi:pulumi:Stack cerulean-cloud-images-test  
Diagnostics:
 docker:index:Image (cerulean-cloud-images-test-cr-orchestrator-image):
   Building your image for linux/amd64 architecture.
   To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

 docker:index:Image (cerulean-cloud-images-test-cr-offset-tile-image):
   Building your image for linux/amd64 architecture.
   To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

 docker:index:Image (cerulean-cloud-images-test-cr-tipg-image):
   Building your image for linux/amd64 architecture.
   To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

Resources:
   ~ 3 to update
   2 unchanged

@github-actions GitHub Actions
Copy link

🍹 preview on cerulean-cloud/test

Pulumi report 
   Previewing update (test):

@ previewing update.....
   pulumi:pulumi:Stack cerulean-cloud-test running 
@ previewing update....
   pulumi:providers:docker cerulean-cloud-images-test-gcr  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-ais  
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cloud-tasks-ais-analysis  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-ais  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-ais create replacement [diff: ~detectMd5hash,name,source]
   gcp:sql:User cerulean-cloud-test-database-users  
   gcp:projects:IAMMember cerulean-cloud-test-cf-ais-iam  
   gcp:sql:Database cerulean-cloud-test-database  
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-ais update [diff: ~environmentVariables,secretEnvironmentVariables,sourceArchiveObject]
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-ais-invoker  
@ previewing update..........
   gcp:serviceaccount:Account cerulean-cloud-test-cr-offset-tile  
++ docker:index:RemoteImage cerulean-cloud-images-test-remote-offset create replacement [diff: ~pullTriggers]
+- docker:index:RemoteImage cerulean-cloud-images-test-remote-offset replace [diff: ~pullTriggers]
++ docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator create replacement [diff: ~pullTriggers]
+- docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator replace [diff: ~pullTriggers]
++ docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg create replacement [diff: ~pullTriggers]
+- docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg replace [diff: ~pullTriggers]
   gcp:projects:IAMMember cerulean-cloud-test-cr-offset-tile-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-offset-tile-secretmanagerSecretAccessor  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-offset-tile-secret-accessor-binding  
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cr-offset-tiles update [diff: ~metadata,template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-offset  
@ previewing update......
   aws:iam:Role cerulean-cloud-test-lambda-titiler-role  
   aws:s3:Bucket cerulean-cloud-test-titiler-lambda-archive  
   aws:iam:Policy cerulean-cloud-test-lambda-titiler-policy  
   aws:sns:Topic cerulean-cloud-test-lambda-APIAbuseAlert  
   aws:apigatewayv2:Api cerulean-cloud-test-lambda-titiler-api  
   gcp:serviceaccount:Account cerulean-cloud-test-cr-orchestrator  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-sr  
   gcp:serviceaccount:Account cerulean-cloud-test-cr-tipg  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cr-orchestrator  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-sr  
   aws:iam:Role cerulean-cloud-test-lambda-sentinel1-iam  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment2  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-support  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jona  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-aemon  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jason  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudTasksEnqueuer  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-secretmanagerSecretAccessor  
   gcp:projects:IAMMember cerulean-cloud-test-cf-sr-iam  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-secretmanagerSecretAccessor  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-sentinel1-basic-execution  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr create replacement [diff: ~detectMd5hash,name,source]
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-orchestrator-secret-accessor-binding  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-tipg-secret-accessor-binding  
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run create replacement [diff: ~detectMd5hash,name,source]
   pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cr-tipg update [diff: ~metadata]
~  gcp:cloudrun:Service cerulean-cloud-test-cr-orchestrator update [diff: ~template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-tipg  
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-orchestrator  [diff: ~service]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-sr update [diff: ~secretEnvironmentVariables]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cf-historical-run update [diff: ~secretEnvironmentVariables]
   aws:lambda:Function cerulean-cloud-test-lambda-sentinel1-sub  
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-sr-invoker  [diff: ~cloudFunction]
   gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cf-historical-run-invoker  [diff: ~cloudFunction]
   aws:sns:TopicSubscription cerulean-cloud-test-sentinel1-subscription  
   aws:lambda:Permission cerulean-cloud-test-lambda-sentinel1-permission  
@ previewing update........................................................................................................................
~  aws:s3:BucketObject cerulean-cloud-test-titiler-lambda-archive update [diff: ~source]
~  aws:lambda:Function cerulean-cloud-test-lambda-titiler-sentinel update [diff: ~sourceCodeHash]
   aws:apigatewayv2:Integration cerulean-cloud-test-lambda-titiler-integration  
   aws:cloudwatch:MetricAlarm cerulean-cloud-test-lambda-titiler-alarm  
   aws:lambda:Permission cerulean-cloud-test-lambda-titiler-permission  
   aws:apigatewayv2:Route cerulean-cloud-test-lambda-titiler-route  
   aws:apigatewayv2:Stage cerulean-cloud-test-lambda-titiler-stage  
   pulumi:pulumi:Stack cerulean-cloud-test running Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   pulumi:pulumi:Stack cerulean-cloud-test running Checking Docker is available...
   pulumi:pulumi:Stack cerulean-cloud-test running Building container image...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully built container image with id sha256:99c6750386f14d96906508ce006250ab73be22bf7ec2af949fc27995e13b071a
   pulumi:pulumi:Stack cerulean-cloud-test running Creating installation package.zip ...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip
-- docker:index:RemoteImage cerulean-cloud-images-test-remote-offset delete original [diff: ~pullTriggers]
-- docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator delete original [diff: ~pullTriggers]
-- docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator delete original [diff: ~pullTriggers]
-- docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg delete original [diff: ~pullTriggers]
-- docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator delete original [diff: ~pullTriggers]
   pulumi:pulumi:Stack cerulean-cloud-test  9 warnings; 6 messages
Diagnostics:
 pulumi:pulumi:Stack (cerulean-cloud-test):
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
   warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.

   Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   Checking Docker is available...
   Building container image...
   Sucessfully built container image with id sha256:99c6750386f14d96906508ce006250ab73be22bf7ec2af949fc27995e13b071a
   Creating installation package.zip ...
   Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip

Resources:
   ~ 8 to update
   +-6 to replace
   14 changes. 53 unchanged

@alukach alukach changed the title IN PROGRESS: Optimization - deterministic deployments Optimization - deterministic deployments Dec 24, 2024
@alukach alukach mentioned this pull request Dec 24, 2024
Draft
@alukach alukach marked this pull request as ready for review December 24, 2024 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@alukach